Wide-field nanosecond imaging methods using wide-field optical modulators

ABSTRACT

Improved resolution of a time-varying optical image is provided with a wide field optical intensity modulator having a bandwidth greater than that of the detector array(s). The modulator configuration can have high photon collection efficiency, e.g. by using polarization modulation to split the incident light into several timegated channels.

FIELD OF THE INVENTION

This invention relates to providing improved time resolution in opticalimaging.

BACKGROUND

It is often desired to provide improved time resolution in imagingoptical measurements. E.g., in fluorescence spectroscopy, fluorescencelifetime provides valuable information. However, typical fluorescencelifetimes are on the order of nanoseconds, which is much too fast fortypical imaging detector arrays. Conventional approaches to this issuetend to require a time-consuming scanning approach using asingle-element fast detector to follow the time dependence of theincident light. This need to provide information on time dependence ofincident light distinguishes this technology from mere fast shutteringof a scene, as in conventional photography. Accordingly, it would be anadvance in the art to provide improved time resolution in imagingoptical measurements.

SUMMARY

We have found that a wide field optical intensity modulator can have abandwidth greater than that of typical optical detector arrays, and cantherefore be used to provide improved time resolution in opticalimaging. In preferred embodiments, the modulator configuration can havehigh photon collection efficiency (the only losses being small parasiticlosses) and may be compatible with standard, inexpensive camera sensors.This combination of benefits makes it especially beneficial forfluorescence lifetime imaging (FLIM), where signals are typically weakand where high photon throughput and rapid acquisition is desired.However, numerous other applications are also possible, as described indetail below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows an embodiment of the invention.

FIG. 1B shows calculated gating efficiency for an exemplary modulator.

FIGS. 2A-C provide experimental results relating to multi-label FLIM.

FIGS. 3A-C provide experimental results relating to wide-field FLIM.

FIGS. 4A-C provide experimental results relating to fast FLIM with athin Pockels cell modulator.

FIGS. 5A-D relate to a multi-pass cavity to provide time binning.

FIGS. 6A-B provide calculated results on lifetime estimation error.

FIG. 7A shows an example of wavelength dependence of gate efficiency.

FIG. 7B shows an example of the result of adding a quarter wave plate toa modulator configuration.

FIGS. 8A-E show various modulator configurations.

FIGS. 9A-B show exemplary time-resolved hyperspectral configurations.

FIGS. 10A-B show a lock-in configuration.

FIGS. 11A-C show an application to charged particle detection.

DETAILED DESCRIPTION

Section A of this description describes general principles relating toembodiments of the invention. Section B is a detailed example relatingto fluorescence lifetime imaging (FLIM) using Pockels cell modulators.Section C describes several further variations, embodiments andapplications. In general, embodiments of the invention are notrestricted to the FLIM application of the example of section B, or tothe use of Pockels cells as in the example of section B.

A) General Principles

As indicated above, the main idea is to use a wide field opticalmodulator that is faster than the camera to provide improved resolutionof time-varying waveform parameters on a pixel-by-pixel basis.

More specifically, an embodiment of the invention is an apparatus forproviding time-resolved optical imaging. The apparatus includes a widefield optical intensity modulator (e.g., combination of 102, 104, 106,108 on FIG. 1A), one or more 2-D detector arrays (e.g., 142, 144, 146,148 on FIG. 1A), and imaging optics (e.g., 118, 120 on FIG. 1A)configured to image incident light onto the 2-D detector array(s)through the wide field optical intensity modulator. The temporalbandwidth of the optical modulator is greater than a temporal pixelbandwidth of the 2-D detector array(s). The apparatus further includes aprocessor (e.g., 130 on FIG. 1A) configured to automatically determineone or more waveform shape parameters of the incident light by analyzingsignals from the 2-D detector array(s) vs. an input modulation appliedto the optical intensity modulator. Here the one or more waveform shapeparameters of the incident light are determined on a pixel-by-pixelbasis of the one or more 2-D detector arrays.

Here ‘waveform shape parameters’ is defined to include threepossibilities: 1) curve fitting parameters such as an exponential decayconstant of received pulses, 2) data points that provide a discretelysampled estimate of a received waveform pulse shape, and 3) parametersof a periodic received signal, such as phase shift and amplitudemodulation, relative to a periodic excitation provided to the scenebeing imaged. Time delay is not a waveform shape parameter because timedelay of a waveform does not result in any change of its shape. Anotherway to see this distinction is to note that an isolated pulsed waveform(e.g., as used in conventional LIDAR (LIght Detection and Ranging)) doesnot have a defined phase.

A detector array is a 2-D array of contiguous optical detector elements.In embodiments with multiple detector arrays, these arrays can beintegrated on the same substrate or they can be separate devices. ‘Widefield’ in this work refers to the optical modulator (intensity orpolarization) having a sufficiently wide aperture to match the 2-Ddetector array. In other words, light received at every pixel of thedetector array is modulated by a single wide-field optical modulator.

In preferred embodiments, optical intensity modulation is provided by apolarization modulator combined with polarizing optics. Variousconfigurations are possible. A first modulator configuration is wherethe wide field optical intensity modulator includes a wide field opticalpolarization modulator disposed between a first polarizer and a secondpolarizer so as to convert polarization modulation to intensitymodulation (example of FIG. 8A).

A second modulator configuration is where the wide field opticalintensity modulator includes an input polarizer followed by a wide fieldoptical polarization modulator followed by a polarizing beam splitter.Here the polarizing beam splitter provides a first output to a first ofthe 2-D detector arrays and provides a second output to a second of the2-D detector arrays. Here also polarization modulation is converted tointensity modulation of the first and second outputs (example of FIG.8B).

An example of the use of the second modulator configuration is where thewaveform shape parameters include an exponential decay time, and wherethe input modulation is a step function. Here the exponential decay timecan be determined by analysis of single-frame signals from correspondingpixels of the first 2-D detector array and the second 2-D detectorarray.

A third modulator configuration is as shown on FIG. 1A and described indetail in section B. Here the wide field optical intensity modulatorincludes:

a) an input polarizing beam splitter (e.g., 102 on FIG. 1A) having afirst output and a second output;

b) a wide field optical polarization modulator (PM) (e.g., 104 on FIG.1A) configured to receive the first and second outputs in parallel andto provide corresponding first and second PM outputs;

c) a first output polarizing beam splitter (e.g., 106 on FIG. 1A)configured to receive the first PM output and to provide a third outputand a fourth output (e.g., outputs 1 and 4 on FIG. 1A); and

d) a second output polarizing beam splitter (e.g., 108 on FIG. 1A)configured to receive the second PM output and to provide a fifth outputand a sixth output (e.g., outputs 2 and 3 on FIG. 1A).

Here the third output is provided to a first of the 2-D detector arrays(e.g., 142 on FIG. 1A); the fourth output is provided to a second of the2-D detector arrays (e.g., 148 on FIG. 1A); the fifth output is providedto a third of the 2-D detector arrays (e.g., 144 on FIG. 1A); and thesixth output is provided to a fourth of the 2-D detector arrays (e.g.,146 on FIG. 1A).

A polarizing element or beamsplitter in any of these modulatorconfigurations may take many forms. These include plate polarizers, thinfilm polarizers, wire-grids, beam-splitting cubes, and polarizingprisms. Some of these may have an in-line configuration such asbirefringent beam-displacers, Rochon, or Wollaston prisms. Apolarization conversion system may be used to convert unpolarized lightto a defined polarization with minimal optical loss while preserving animage. Such a system would be especially suited as a first polarizingelement to increase photon efficiency in cases where there only a singlebeam is modulated. A final possibility includes having spatiallyseparated regions of an array detector each with a different polarizingelement in front of the sensor. Similarly, each pixel of the arraydetector may have its own polarizing element. Such an integratedconfiguration as found in polarization camera sensors removes the needfor image registration of beamsplitter outputs.

The input modulation can be a pulse having an automatically adjustabletime delay td after an optical excitation provided to a scene. Here theone or more waveform shape parameters can include data points ofdetector array signals vs. time delay.

The input modulation can be selected from the group consisting of: astep function, a sampling pulse, and periodic modulation for lock-indetection.

The wide field optical intensity modulator can include a longitudinalPockels cell having a direction of optical propagation and an appliedelectric field direction that coincide. Such longitudinal modulators usepotassium dideuterium phosphate (KD₂PO₄-DKDP) or potassium dihydrogenphosphate (KDP) crystals. This configuration tends to be moreappropriate for many applications than a transverse Pockels cellconfiguration. Optical intensity modulators may also include standardtransverse electric field Pockels cell configurations having largeaperture. These may be ideal for systems requiring a resonant highvoltage drive or larger acceptance angle. Standard commerciallyavailable transverse modulators involve two crystals rotated by 90degrees or separated by a half-wave plate in such a way as to canceloff-axis birefringence effects. This improves their imaging performanceand also thermal stability. Such dual modulators are available withapertures greater than 10 mm in standard materials including rubidiumtitanyl phosphate (RbTiOPO₄) and lithium tantalate (LiTaO₃).

The imaging optics can include a multi-pass optical cavity having acavity round trip time, where the multi-pass optical cavity isconfigured to provide optical time resolution according to multiples ofthe cavity round trip time (example of FIGS. 5A-D). The modulator inthis case may be either internal or external to the multi-pass imagingoptics.

The incident light can be a periodic signal that is responsive to aperiodic excitation of a scene being viewed. Here the wide field opticalintensity modulator is preferably resonantly driven synchronously withrespect to the periodic signal (example of FIGS. 10A-B).

Here a modulator is driven synchronously with respect to a periodicsignal in the incident light if the modulation frequency is the same asthe frequency of the incident light (homodyne). For the homodyne case,the modulation frequency is phase locked (or otherwise held in aconstant phase relationship) with respect to the frequency of theincident light. The heterodyne case is also of interest.

The optical intensity modulator can include two or more opticalmodulators having identical or different input modulation signals.

The imaging optics is typically configured to view a scene. Here a‘scene’ being viewed is any combination of one or more objects as viewedthrough an optical imaging system of any kind (e.g., microscope,endoscope, telescope, etc.). Excitation of such a scene can be providedby various excitation methods (optical, electrical, magnetic, etc.). Inthis work, the main signal of interest is optical radiation from thescene in response to the excitation. In many cases, this opticalradiation from the scene is a nonlinear response of the scene to theexcitation. More specifically, such a nonlinear response has frequencycomponents in the response that are not present in the excitation, e.g.as in optical fluorescence. Equivalently, for optical excitation, such anonlinear response has wavelength components in the response that arenot present in the excitation.

An optical response of the scene to an excitation can provide theincident light. In many cases of interest, the optical response of thescene is a nonlinear response. The optical response may also result in achange of the shape of the incident light waveform compared to theillumination waveform. The wide-field optical intensity modulator can bedriven with a modulation signal having a controllable delay after theexcitation.

In cases where output polarizing beam splitters are used, the outputsare complementary. E.g., if one output is modulated according to anapplied modulation signal G(t) the other output is modulated accordingto 1-G(t).

B) Experimental Demonstration

B1) Introduction

Existing sensors for wide-field nanosecond imaging sacrifice performanceto gain temporal resolution, failing to compete with scientific CMOS andelectron-multiplying CCD sensors in low-signal applications. A varietyof detectors currently access the nanosecond regime. Gated opticalintensifiers (GOIs) based on microchannel plates (MCPs) allow forsub-nanosecond gating in a single image frame, and segmented GOIs canacquire multiple frames when combined with image splitting. Gating inton frames in this way limits overall collection efficiency to <1/n, andperformance is further limited by photocathode quantum efficiency, MCPpixel density, excess noise, and lateral electron drift. Streak cameratechniques have also been demonstrated for widefield imaging, but theyalso require a photocathode conversion step and additional high-lossencoding. Single-photon avalanche detector (SPAD) arrays are an emergingsolid-state approach, but they are currently limited to sparse fillfactors and high dark currents.

The limitations of current nanosecond imaging techniques areparticularly manifest in fluorescence lifetime imaging microscopy(FLIM). Fluorescence lifetime is a sensitive probe of local fluorophoreenvironment and can be used to report factors like pH, polarity, ionconcentration, Förster resonance energy transfer (FRET), and viscosity.As lifetime imaging is insensitive to excitation intensity noise,labelling density, and sample photobleaching, it is attractive for manyapplications. FLIM typically relies on confocal scanning combined withtime-correlated single photon counting (TC-SPC) detectors. Thethroughput of TC-SPC is limited by the detector's maximum count rate(typically 1-100 MHz), and confocal microscopy relies on high excitationintensities that can cause non-linear photodamage to biological samples.Frequency domain wide-field approaches are a promising alternative, butthey currently require demodulation with either a GOI or high-noisemodulated camera chip. Given the disadvantages of existing wide-fieldand TC-SPC approaches, FLIM especially calls for the development of new,efficient imaging strategies to extend its utility for bio-imaging.

Here we demonstrate ultrafast imaging techniques—compatible withstandard cameras—that can have no inherent loss or dead time, allowingaccess to subframe rate sample dynamics at timescales as fast asnanosecond fluorescent lifetimes. First, we show an all-photonwide-field imaging system based on polarizing beam-splitters (PBS) and aPockels cell (PC). This can be used to create two temporal bins or tomodulate images on any timescale—from nanoseconds to milliseconds. Weuse this to demonstrate efficient wide-field FLIM of a multi-labelledsample, single molecules, and a biological benchmark. Second, wedemonstrate the use of a re-imaging optical cavity as a time-to-spaceconverter to enable n-frame ultrafast imaging when combined with aPockels cell gate.

B2) Results

B2a) Gating with Two Temporal Bins

FIGS. 1A-B show wide-field efficient ultrafast imaging with a Pockelscell. FIG. 1A is a schematic of two temporal bin wide-field imaging fora single pixel fluorescence decay. Fluorescence emission is firstpolarized by PBS 102, a time dependent retardance (step functionillustrated) is applied by the PC 104, and polarizations are split againbefore the sensor by PBS 106 and PBS 108. Two pairs of outputscorrespond to integrated intensity before (1, 3) and after (2, 4) a stepfunction gate is applied in the illustration. Outputs 1, 2, 3, 4 arereceived by detector arrays (or cameras) 142, 144, 146, 148,respectively. Here relevant time dependences are shown in inset 110,where 112 is the excitation, 114 is the fluorescence, and 116 is themodulator signal. The imaging optics is schematically shown as 118 and120. The processor as described above is referenced as 130. Forconvenience, the imaging optics and processor are not shown onsubsequent figures, although they are generally present. Connectionsbetween processor 130 and other parts of the system are also not shown.Practice of the invention does not depend critically on the imagingoptics, and any known imaging configuration can be employed (e.g.,microscope, telescope, endoscope, etc.). Equal optical path lengths areused in practice, and these paths may incorporate standard imagingoptics such as beam-expanders, mirrors, and relay optics. FIG. 1B showsgating efficiency (I_(N)−I₀) calculated for a 30 mm KD*P Pockels cell asa function of incident angle from conoscopic interference patterns,demonstrating high efficiency gating for wide-field imaging within 6mrad half-acceptance angle.

Light from an imaging system is polarized with a beam-splitter, and theimage associated with each polarization is aligned to propagate throughdifferent locations in a wide-aperture PC, as shown in FIG. 1A. The PCprovides an electric field-dependent retardance between the inputlight's polarization components, mapping the temporal signature of theapplied field onto the polarization state of the imaging beams. A secondPBS after the PC again splits the separated imaging beams, giving fourimage frames on the camera. The resulting images now encode temporalinformation, as shown in FIG. 1A. To illustrate our method, we considera step function voltage pulse applied at delay time t_(d) with respectto a short (˜ns) excitation pulse. The step function with edge at tdcreates pairs of output images corresponding to integrated signal beforeand after td. All photons, both before and after the gate, may becollected in a single camera exposure.

In practice, we implement this configuration with either a Gaussiangating pulse at td or a step gate with few nanosecond rise time asdescribed in the following examples. In fact, arbitrary V(t) may beapplied to the PC for specific applications (see Discussion). Note thata gating pulse can be applied either as a single shot measurement orover repeated events integrated in one camera frame. Fluorescencelifetime may be recovered by either varying the gate delay t_(d) todirectly measure the fluorescence decay (see multilabel FLIM below) orby single-frame ratios of gated and ungated channel intensities (seesingle-molecule FLIM below). In cases where the PC aperture is limited,two separate PC crystals may be used instead of using different areas ofthe same crystal. Separate gates can be applied to each PC to createfour time bins as shown, for example, in FIG. 8C.

B2b) Imaging Through Pockels Cells

An important aspect of this technique was realizing that Pockels cellsmay be ideally suited to wide-field imaging. For decades, Pockels cellshave been ubiquitous in applications like pulse-picking, Q-switching,and phase modulation. However, the most common Pockels cellsconfigurations in use are not suited to wide-field imaging.Specifically, they typically have either a small aperture for transversefield modulators or a narrow acceptance angle of a few milliradian forlongitudinal modulators. This severely restricts either field-of-view ornumerical aperture in imaging applications.

For example, standard PCs often use thick (30-50 mm) potassiumdideuterium phosphate (KD*P) crystals with longitudinal field. Thesegive high extinction ratios and are ubiquitous for Q-switching and phasemodulation applications. Off-axis rays experience different birefringentphase shifts than those on-axis, limiting the numerical aperture (NA) ofthe crystal for wide-field imaging. In an image plane, the PC halfangular acceptance a limits the NA of collection optics to Mα for smallangles, where M is magnification. In a diffraction plane (or infinitycorrected space), the field of view (FOV) is instead limited to 2tan(α)f_(obj) where f_(obj) is the imaging objective focal length. Forexample, a 10 μm FOV may be achieved with a 1.4 NA microscope objective(f_(obj)=1.8 mm) and 40 mm thick longitudinal KD*P PC crystal in theinfinity space (α˜4 mrad). FOV can be further improved by magnifying thebeam until the PC aperture becomes limiting. Conventional KD*P PCs arelimited to long pulse repetition rates in the 10's of kHz bypiezoelectric resonances. We note that ultimate repetition rate dependson high voltage pulse shape and crystal dimensions. Electro-optic pulsepickers can operate to 100 kHz and even into MHz rates with low-piezomaterials. Further, periodic drive avoids exciting piezoelectricresonances and is compatible with frequency-domain FLIM at highexcitation rates.

To assess gating efficiency, the impact of off-axis birefringence wassimulated using Mueller matrices and the index ellipsoid of the crystalto arrive at a conoscopic interference (isogyre) pattern, as viewedthrough crossed polarizers. Subtracting the transmitted intensitypattern I at zero voltage (V₀) from that at the half-wave voltage(V_(π)) gives the gating efficiency (I_(π)−I₀), where the useful NA ofthe PC is set by the region of high gating efficiency at lower angles(FIG. 1B). The PC is treated as a linear homogeneous retarder withoff-axis retardance determined by a coordinate transformation of thecrystal axes. We performed this analysis to identify idealconfigurations of standard Pockels cells for wide-field imaging. Inlongitudinal crystals, angular acceptance may be improved by making thecrystal thinner, with a 3 mm crystal increasing α to ˜20 mrad,effectively removing NA and FOV restrictions for microscopy. Here weshow results using a thick 40 mm commercial PC (FIGS. 2A-C and 3A-C) anda custom 3 mm KD*P PC (FIGS. 4A-C). Further, complete zero-fieldcancellation of off-axis birefringence may be obtained by combining thenegative uniaxial (n_(e)<n_(o)) KD*P crystal with a positive uniaxial(n_(e)>n_(o)) static compensating crystal (e.g. MgF₂ or YVO₄). Such acrystal fully compensates for off-axis rays at V₀ and further improvesthe NA at V_(π) (KD*P becomes biaxial with applied field, preventingfull high voltage compensation). This strategy is especially promisingfor thin KD*P crystals being used as a gate inside of a multi-passoptical cavity where the PC remains off for many passes of the crystal.

We have found that Pockels cells may have even larger acceptance anglesby using industry standard dual-crystal compensated, transverse fielddesigns. Here off-axis birefringence and thermal effects can be removedby having two transverse electro-optic modulators either rotated 90degrees relative to each other or having a half-wave plate between them.This effectively exchanges the ordinary and extraordinary rays whilealso switching the electric field direction, cancelling off-axisbirefringence effects and thermal birefringence effects. Suchdual-crystal modulators are known to provide large acceptance angles. Infact, theoretically perfect off-axis cancellation for imagingapplications may occur in modulator units where the optical axis of theelectro-optic crystals and their propagation axis are perpendicular.Typical dual-crystal modulators have very small apertures, but they areavailable commercially with apertures>10 mm in materials like rubidiumtitanyl phosphate and lithium tantalate, requiring proportionally higherswitching voltages.

Thin DKDP crystal modulators are less commonly found, but they may beconstructed by combining the thin crystal substrate with suitableconducting and optically transparent electrodes such as glass coatedwith indium tin oxide or other transparent conductive coatings,conductive transparent films, wire meshes, optical micro-meshes, etc.

Driving electronics for the Pockels cell may include any high voltagewaveform generator or amplifier including for example avalanchetransistors, MOSFET stacks, high voltage MOSFETS in half or full-bridgeconfigurations, drift step recovery diodes, flyback or resonanttransformers, pulse forming networks, or non-linear or saturabletransmission lines. For resonant configurations, RF drives may beimpedance matched to a resonant tank circuit containing the Pockels cellas an electrical component. Such circuits may contain standard L,R,Celements, impedance matching networks, or also resonant transmissionlines or transformers for example. Cooling provisions may be provided tocounteract dielectric and/or resistive heating. Dielectric fluids may beused to prevent high-voltage breakdown, match refractive indices, or toprovide cooling to the crystal.

B2c) Multi-Label FLIM

FIGS. 2A-C relate to Multi-label FLIM. FIG. 2A shows direct measurementof fluorescence decays obtained by sweeping gate delay time td fororange (0, 4.9 ns), red (R, 3.4 ns), nile red (NR, ˜3.1 ns), infrared(IR, 2.3 ns) and propidium iodide (PI, 14 ns) beads. Fitted decayconstants τ are given. The measured Gaussian instrument responsefunction (IRF) is also plotted. FIG. 2B is an intensity image of athree-label wide-field sample of orange, nile red, and infrared beads(labels strongly overlap spatially). FIG. 2C is a lifetime image thatreveals spatial distribution of the labels. Lifetime is measured byfitting the decay traces at each pixel (scale bars 10 μm).

The two bin method has no intrinsic gating loss and allows for imagingonto any sensor. Fluorescence lifetime imaging is thus an idealdemonstration for the technique, where the PC gating pulse is appliedafter delay td from the fluorescence excitation. Lifetime may then bedetermined by either varying the delay time td over multiple frames (asused here) or by taking the single-frame ratio of pre- and post-gateintensities (following section). In FIGS. 2A-C we image a mixture ofthree labels having different lifetimes measured individually to be 3.1ns (2 μm nile red Invitrogen beads), 4.9 ns (0.1 μm orange Invitrogenbeads—background), and 2.3 ns (0.1 μm infrared Invitrogen beads—formedinto crystals). For this data, the PC was located in the image plane,allowing for wide-field FLIM of bright samples at 0.1 NA and 20×magnification with 100 micron FOV. The sample is excited by laser pulseswith duration 1 ns at 532 nm and 5 kHz repetition rate. The fluorescencesignal results from the convolution of the decay function with thelaser's Gaussian excitation pulse with FWHM pulse width ˜2.4 σ_(e). Thecommercial PC used in FIGS. 2A-C applies a Gaussian gate functiong(t−t_(d)) in our experiment with a pulse width of 2.6 ns. By sweepingthe delay time t_(d), the convolution of the fluorescence with thegating function is measured: f(t,τ,σ_(e))*g(t−t_(d)). Temporalinformation such as fluorescence lifetime may be calculated by directlyfitting the measured convolution. Note that the convolution ofexcitation and gating functions in this case gives a Gaussian instrumentresponse function (IRF) with σ_(IRF)=σ_(e) ²+σ_(g) ², measured directlyin FIG. 2A. The fitting approach samples the fluorescence decay at moretime points and can be advantageous for brightly labeled samplescompared to a two-bin measurement. This could be used to moreeffectively measure multi-exponential decays for instance.

B2d) Wide-Field FLIM of Single Molecules

FIGS. 3A-C relate to Wide-field FLIM of Alexa Fluor 532 molecules. FIG.3A show gated channel intensity. FIG. 3B shows ungated channel intensity(scale bar 1 μm). FIG. 3C shows measured lifetimes plotted along withtotal brightness for the numbered, diffraction limited regions with SEerror bars indicated. The majority of these spots are single-moleculeemitters as demonstrated by their photobleaching and blinking dynamics.

For signal-limited applications relying on efficient photon collectionor requiring fast acquisition rates, fluorescence lifetime is bestdetermined by the ratio of gated and ungated intensity in a singleframe. In FIGS. 3A-C, we demonstrate wide-field lifetime microscopy ofAlexa Fluor 532 molecules on glass in a 10×10 μm region. The measuredlifetimes are consistent with both the ensemble lifetime of 2.5 ns andthe large molecular variation seen in similar studies on glass. The PCis used in the infinity space of the microscope objective to apply thesame Gaussian gating function at t_(d)=1.6 ns and 15 kHz repetitionrate. The ratio of the gated and ungated intensity is given by

$\begin{matrix}{R = {\frac{\int{{g\left( {t - t_{d}} \right)}{f\left( {t,\tau,\sigma_{e}} \right)}{dt}}}{\int{{{f\left( {t,\tau,\sigma_{e}} \right)}\left\lbrack {1 - {g\left( {t - t_{d}} \right)}} \right\rbrack}{dt}}} = {\frac{g*f}{\left( {1 - g} \right)*f}❘_{t = t_{d}}}}} & (1)\end{matrix}$

To calculate lifetime, this ratio is experimentally determined bysumming intensity in a region of interest around each molecule. Thisapproach allows single-molecule lifetime spectroscopy while maintainingdiffraction limited resolution and efficient photon collection of ˜7×10³photons per molecule (15 s exposure time). FIG. 3C shows the estimatedlifetime and total brightness for each numbered diffraction-limitedemitter along with error-bars for the lifetime estimation. Estimation islimited by fluorescence background and dark current here. A low-costindustry CMOS machine vision camera (FLIR) is used for the detector. Inthis case, the angular acceptance of the PC limits the field of view to10 μm but still allows photon collection at 1.4 NA. Single-moleculelifetime spectroscopy in wide-field remains challenging with confocalapproaches, whereas here it is readily demonstrated with PC gating andan inexpensive, high-noise camera.

B2e) Fast FLIM with a Thin PC

FIGS. 4A-C relate to Fast FLIM with a thin PC. FIG. 4A is an intensityimage of Convallaria majalis rhizome stained with acridine orange, astandard FLIM benchmark (scale bar 100 μm). FIG. 4B is a lifetime imagefrom fitting a timing trace of 100 ms exposures (50 μW excitation). FIG.4C is a lifetime image from a single 100 ms acquisition frame. The inset(lower right of FIG. 4C) demonstrates the same single frame with 2 msexposure at high excitation intensity (3 mW—high intensity significantlyreduced lifetime in this sample, possibly due to photochemistry).Lifetime images include an intensity mask to show sample structure.

By using a thin PC crystal, these techniques are extended to ultra-widefields of view. A 3 mm thick KD*P Pockels cell with a 20 mm aperturegates nearly the entire output of a standard inverted microscope with an0.8 NA objective. A 4.5 ns rising edge pulse was used at 5 kHzrepetition rate to image a standard FLIM benchmark in FIGS. 4A-C. Singleframe and trace-fitting analysis demonstrate rapid acquisition ofmegapixel FLIM images with 300 μm square FOV. Single frame exposures of100 ms and 2 ms are demonstrated—the latter may be taken at the maximumcamera frame rate. These acquisitions show dramatic throughput advantagefor wide-field acquisition. The 100 ms frame in FIG. 4C is formed fromsingle exposure detection of 4.8×10⁸ photons by 0.8 megapixels. Theinset in FIG. 4C (lower right) likewise corresponds to a 2 ms exposureof the same frame at higher excitation intensity with 3.1×10⁸ detectedphotons. The number of detected photons is inferred from the knownresponsivity of the camera. Compared to a 2 MHz TC-SPC photon countingrate (standard to avoid pile-up error), these acquisitions give a 2,400×and 78,000× enhancement in photon throughput respectively. Such highthroughput and potential for low exposure times will enable future FLIMstudies on dynamic samples. The single-frame acquisition method isparticularly powerful, as it prevents image motion artifacts (caused bymultiple acquisition frames or scanning for example) and allowsself-normalization within a single exposure to remove intensity noise.Quantitative lifetimes are easily calculated using the pre-calibratedIRF as described in the prior sections.

B2f) Gated Re-Imaging Cavities for Multi-Frame Imaging

FIGS. 5A-D relate to multi-frame nanosecond imaging with a cavitytime-to-space converter. FIG. 5A shows an externally gated, tiltedmirror 4f re-imaging cavity. Image input is on small in-coupling mirrorM1 in an image plane (i). M2 is tilted at a diffraction plane (f),spatially offsetting the images at the M1 plane each pass. Each roundtrip, images are passively out-coupled through partially transmissivemirror M2. Here 502 and 504 are the re-imaging lenses for the 4f cavity,and 506 is the PC output gate. FIG. 4B show normalized image intensitiesfor four output images from the cavity showing a 4 ns round trip delay.The cavity output on camera (CMOS camera) images of FIG. 5C show fourimages output from the PC analyzer for each round trip output from thecavity (columns numbered 1-4 as in FIG. 1A). Four round-trips (rowsi-iv) are displayed (scale bar 50 μm). The sample is a mixture ofdrop-cast nile red 2 μm beads (˜3.1 ns) and orange 0.1 μm beads (4.9 ns)that form the diffuse filaments. FIG. 5D shows the ratio of outputframes (row i, column 4) and (row ii, column 4) in the gated channel att_(d) ⁼5 ns (red line in (b)) that is used for single frame FLIM asdescribed below. The two labels are readily differentiated (scale bar 10μm).

Nanosecond imaging with PCs can be extended beyond two temporal binsthrough the use of gated re-imaging optical cavities. Larger bin numbersenable increased estimation accuracy for multi-exponential decays,improve lifetime dynamic range, and also allow efficient single-shotultrafast imaging. We exploit the round-trip optical delay of are-imaging cavity combined with a tilted cavity mirror to providenanosecond temporal resolution by spatially separating the cavity roundtrips. While imaging with n-frames using GOIs is limited to <1/ncollection efficiency, this re-imaging cavity technique enablesefficient photon collection for low-light or single-photon sensitiveapplications. In related work, cavities have been used for singlechannel orbital angular momentum and wavelength to time conversion.Aligned optical cavities have been used for time-folded optical imagingmodalities like multi-pass microscopy. Our implementation insteademploys a re-imaging cavity as the means to obtain temporal resolutionfor wide-field imaging.

An image is in-coupled to a 4f cavity at the central focal plane bymeans of a small mirror M1 as shown in FIG. 5A. The 4f configurationre-images the end mirrors (diffraction planes) every round trip. If oneend mirror M2 is tilted by angle θ, the image position y_(i) at thecentral focal plane after n round trips is displaced by y_(i)=fsin(2nθ), where f is the focal length of the 4f cavity. The angle θ isset such that the resulting images are not blocked by the in-couplingmirror. Each sequentially displaced image is delayed in turn by anadditional round trip. To extract temporal information, the spatiallyseparated images need to be either gated externally or simultaneouslyoutcoupled from the cavity using a PC. In the externally gated scheme(schematically shown in FIG. 5A), light is passively out-coupled eachround trip through a transmissive mirror. The spatially displaced imageshave a relative time delay Δt=8fn/c based on their number of round tripsn, and an external gate is simultaneously applied to all delayed imagesto create temporally distinct frames. A step function gate V(t) allowslifetime measurement from the ratios of the time-delayed bins, similarto the two-bin case described above. Using the two-bin PC scheme as theexternal gate gives four image frames from each round trip output (FIG.5C). Photon efficiency, the ratio of detected photons to the numberinput to the cavity, with end-mirror reflectivity r is given by 1−r^(n)after n round trips, ignoring intracavity loss. This efficiency can bemade very high for an appropriate choice of r. For example, 87%efficiency is obtained with r=0.6 and n=4. It should be noted that theintensity variation between the different frames is caused by partialtransmission after n round trips.

FIGS. 5C-D demonstrate the output from an externally gated tilted mirrorcavity. Here a Gaussian gate pulse of width less than the round triptime is used. Lifetime in FIG. 5D is calculated from the ratio R of twoframes (FIG. 5B images (row i, column 4) and (row ii, column 4)) in thegated channel delayed by one cavity round trip time t_(rt) of 4 ns asR=(g*f|_(t) _(d) )/(g*f|_(t) _(d) _(+t) _(rt) ). Alternatively, bothgated and ungated frames could be included in the estimation to make useof all photons as in equation (1).

In a second gated cavity scheme, there is instead no transmissivemirror, and all input light is simultaneously outcoupled from the cavitywith an intracavity Pockels cell and polarizing beamsplitter. Morespecifically this configuration may have the pockels cell 506 inside thecavity between elements 502 and 504 with a polarizing beamsplitterelement also between 502 and 504 for out-coupling. Such a schemedirectly gives n images with sequential exposures of t_(rt)=8f/c andleaves no light in the cavity. Either a thin-crystal or compensated PCwould be preferable for intracavity gating since the light passesthrough the PC each round trip. It is interesting to compare n-bin andtwo-bin lifetime methods in terms of their theoretical estimationaccuracy (see FIGS. 6A-B). While the overall accuracies may be closelymatched for monoexponentials, n-bin methods have the advantage of awider temporal dynamic range.

These cavity imaging methods have the advantage of zero dead-timebetween frames and have no inherent limits on collection efficiencybeyond intracavity loss. The externally gated cavity is straightforwardto implement with thick-crystal PCs, but has the disadvantage ofindirect temporal gating. Intracavity gating instead allows for truen-frame ultrafast imaging where each round trip corresponds to onetemporally distinct image frame. Round trip times from 1 to 10 ns may beachieved with standard optics. We note that an alternative approach ton-bin imaging could similarly use multiple two-bin gates in series(e.g., as on FIG. 8D) with the added complexity of multiple PCs anddetectors.

B2g) Theoretical Estimation Accuracy

FIGS. 6A-B relate to lifetime estimation error. FIG. 6A shows theCramer-Rao bound on lifetime estimation accuracy for a monoexponentialfluorescence decay using different numbers of bins. Dashed lines comparetwo to n-bin lifetime measurements in the case where the measurementwindow is n×4 ns. The solid line corresponds to two-bin lifetimeestimation at 4 ns t_(d) without finite measurement window (i.e. idealstep function gate). Note that the range of maximum sensitivity can beshifted with t_(d). The dash-dotted horizontal line indicates the shotnoise limit: σ_(τ)/τ=1/√{square root over (N)}. FIG. 6B shows simulatedlifetime resolution for a realistic two-bin PC experiment with a 1 ns10-90% logistic rise time PC gate and 1 ns σ_(e), similar to the solidline case in FIG. 6A. Near shot noise limited estimation accuracy may beobtained for τ>PC rise time.

Two-bin lifetime estimation can perform surprisingly well when comparedto the Cramer-Rao bound for n-bin TC-SPC. Both two-bin and n-binestimation accuracy scale with photon counting shot noise. FIGS. 6A-Bshow that n-bin measurements have a wider dynamic range of lifetimesensitivity, but that a two-bin PC gate can be nearly as accurate formono-exponential decays when tuned to the appropriate gate delay. TC-SPCgains a large number of temporal bins from the bit depth of the ADC,which dominantly affects the dynamic range. With ideal PC gating,estimation within a factor of two of the shot noise limit (SNL) may beobtained over a decade of lifetimes with peak sensitivity˜1.3×SNL. Infact, for a step function gate with 1 ns PC rise time, estimation within2-3×SNL can be obtained between 1 and 10 ns.

B2h) Spectral Dependence of PC Gating Efficiency

FIG. 7A shows calculated spectral dependence of PC gating efficiencywith V_(π) set for 532 nm. High efficiency gating is shown over a 100 nmwavelength range compatible with fluorescence emission spectra. FIG. 7Bshows intensity of output image channels as a function of PC retardance.In the case shown in FIG. 1A, crossed polarizers result in pairs ofoutput frames whose intensities follow lines 702 and 704. If a quarterwaveplate (QWP) is inserted at one of the outputs of the PC before thesecond PBS, then two of the channels can instead follow lines 706 and708. This has the effect of introducing an optical phase shift betweenoutput images when using a sinusoidal modulation. Addition of a quarterwaveplate in front of or directly after the optical modulator also hasthe effect of shifting the range of modulation necessary for fullswitching, halving the required amplitude for resonant sinusoidal drivesmodulating between 0 and pi phase shifts. We also note that the QWPoutputs are spectrally sensitive near 532 nm, where the gatingefficiency in FIG. 7A is insensitive. This could potentially beexploited for multi-dimensional measurements of spectrum and lifetime.

B3) Discussion

We have presented methods for two and n-bin temporal imaging onnanosecond timescales using Pockels cells. Proof-of-concept experimentswith single molecule lifetime spectroscopy and wide-field FLIMdemonstrate the potential to bring nanosecond resolution tosignal-limited applications. Our approach is photon efficient andretains the sensitivity and image quality of scientific cameras, makingit widely compatible and potentially inexpensive. The ability to performsingle-frame FLIM without gating loss is a particularly uniqueadvantage, as it enables dynamic FLIM without the loss, noise, andpotential motion and intensity artifacts of other approaches. Replacingpoint-scanning FLIM with efficient wide-field acquisition may proveespecially useful in bio-imaging applications such as lifetime FRET,single-molecule and super-resolution microscopy, multi-modal imaging,and clinical diagnostics. Further applications may be found in ultrafastimaging, time-to-space multiplexing, lock-in detection, andtime-of-flight techniques.

For FLIM applications, nanosecond imaging with PCs enables largeimprovements in throughput over conventional TC-SPC. Even at lowrepetition rates, PC FLIM throughput readily surpasses TC-SPC. Forexample, a PC gated image at a low signal level of 1 photon/pixel/pulseat 15 kHz for a 1 megapixel image would take 7,500 times longer toacquire on a 20 MHz confocal TC-SPC system operating at a 10% count rate(standard to avoid pile-up). This throughput advantage grows linearlywith signal and pixel number. Note that PCs may gate 1photon/pixel/pulse without saturation, unlike GOIs or TC-SPC detectors.Wide-field, high throughput lifetime imaging with PCs could enableimaging of biological dynamics at high frame rate. An example of arelevant application would be real-time imaging of cellular signaling,especially in neurons. FLIM may also be applied as a clinical or in vivodiagnostic and wide-field gating may be readily compatible withendoscopic probes.

PC imaging overcomes the limitations of other wide-field technologies.Gated optical intensifiers in particular face technical drawbacksincluding low photocathode quantum efficiency, reduced resolution,multiplicative noise, and saturation. Further, the loss of ungatedphotons (collecting l/n for n temporal bins) necessitates multi-exposureFLIM acquisition. We note that frequency modulated cameras have recentlybeen developed to enable high-throughput FLIM, but these suffer fromvery high dark currents and read noise. PC modulation provides analternative approach to frequency domain FLIM which can also allow MHzexcitation rates.

PC gating may further allow for new microscopy techniques by exploitingthe nanosecond temporal dimension. For example, spectral information hasbeen used to enable multi-labelling of biological samples, which provesimportant in understanding complex intracellular interactions.Fluorescence lifetime may similarly provide an attractive temporalapproach for unmixing multi-labeled signals. Confocal FLIM has alreadybeen applied to this problem. In studying single molecules, thecapability to combine parallel lifetime measurements with spatial andspectral channels could allow for new types of high-throughputspectroscopy experiments to study molecular populations andphotophysical states. New information from lifetime could also be usedto enhance spatial localization in super-resolution microscopy. Further,temporal gating could be used to suppress background autofluorescenceoccurring at short lifetimes.

While we have primarily focused on applications in fluorescencemicroscopy, we also note that PC nanosecond imaging techniques could bemore broadly applied in quantum optics for fast gating, lock-indetection, event selection, or multi-pass microscopy. Other usefuloperation modes may be realized with the two-bin PC scheme by applyingdifferent modulations V (t). Traditional fast-imaging applications inplasma physics, laser-induced breakdown spectroscopy, combustion,time-of-flight techniques, and fluid dynamics could also benefit fromsensitive single-shot imaging. The n-frame tilted mirror re-imagingcavity is particularly unique in its ability to perform single-shotultrafast imaging of weak, non-repetitive events with zero deadtimebetween frames when using an internal PC gate. It could also proveuseful for wide dynamic range lifetime imaging.

In summary, wide-field PC FLIM was demonstrated in single-frame and timetrace modalities. Single-molecule lifetime spectroscopy showedcompatibility with signal limited applications. By using a thin PCcrystal, the technique was extended to ultra wide-field FLIM with singleframe acquisition. FLIM images were acquired on a standard biologicalbenchmark with exposure times down to 2 ms and acquisition speeds to thecamera frame rate. Finally, a new method using re-imaging cavities toenable ultrafast imaging by time-to-space multiplexing was shown. Thesetechniques promise to open the nanosecond regime to signal-limitedapplications like wide-field and single-molecule fluorescencemicroscopy. Further, they are broadly compatible with any imaging systemand sensor, giving potential applications in a variety of fields.

B4) Methods

B4a) Experimental Setup

FLIM was performed with a homemade fluorescence microscope and a thick,commercial PC crystal for FIGS. 2A-C, 3A-C, and 5A-D. A Nikon PlanApo100× VC 1.4 NA oil-immersion objective was used for single-moleculemicroscopy. All other data was taken with a 20×, 0.8 NA Zeiss PlanApoobjective. Excitation pulses (1 ns FWHM) at 532 nm were generated by aQ-switched Nd:YAG at 5 kHz repetition (15 kHz for single-molecule data)(Standa Q10-SH). The detector was a machine vision CMOS camera (FLIRBFS-U3-32S4M-C). A 10 mm aperture, 40 mm thick dual crystal longitudinalKD*P PC embedded in a 50Ω transmission line was used (Lasermetrics1072). This PC uses two crystals (rotated 90 degrees and with oppositeelectric fields), halving the required half-wave voltage. This designgave a comparable acceptance angle to a single-crystal PC of similarthickness (α˜4 mrad). We note that in theory this configuration canachieve some degree of off-axis cancellation with well-matched crystals.High voltage gating pulses were generated into 50Ω with an amplitude of1.3 kV, 2.8 ns FWHM (FID GmbH) attaining 85% of V_(π) and σ_(IRF)=1.1ns. Laser and HV pulser were synchronized with a DG 535 delay generator(Stanford Research Systems). Timing jitter was <100 ps. Longtransmission lines were used to prevent spurious pulse reflectionsduring fluorescence decay. For single-molecule data, only two of theoutput frames (one output from first PBS) were used to maximize FOVthrough the PC, limiting photon efficiency to ˜50%. This is not afundamental limitation of the technique but was used to simplify ourimplementation with a limited single PC aperture. IRFs were acquiredusing a frosted glass sample.

The thin PC crystal demonstration in FIGS. 4A-C was performed on aninverted microscope (Zeiss Axiovert) using a 20×, 0.8 NA Zeiss PlanApoobjective and Andor Neo5.5 sCMOS. A 3 mm thick, 20 mm aperture KD*Plongitudinal PC was home-built along with a high voltage driver capableof supplying nanosecond switching pulses with amplitudes up to 5 kV. Agating efficiency of 0.8 is used with a rise time of 4.5 ns for the datain FIGS. 4A-C. Only one polarization channel is demonstrated here. Bothchannels may be incorporated by adding a second PC or Wollaston prism.

The 4f re-imaging cavity used for the n-bin demonstration used a 3 mmprism mirror (Thorlabs MRA03-G01) for in-coupling and f=150 mm(t_(rt)=8f/c=4.0 ns). Passive out-coupling was through a neutral densityfilter of optical density 1 (R=0.4 and T=0.1). Relay lenses were used tocreate an image plane at the PC and again at the camera (CMOS). Pick-offmirrors combined imaging beams generated by the two PBS with equal pathlengths.

B4b) Sample Preparation

Alexa 532 (Invitrogen, Thermo Fisher) single-molecule samples wereprepared by drop casting dilute solution onto a hydrophobic substrate,then placing and removing a pristine coverslip. A dense field wasphotobleached to the point that single, diffraction-limited emitterswere observed. Step-like photobleaching was observed along with blink-ondynamics. While multi-molecule emission within a diffraction limitedspot was certainly also seen, a majority of the emitters were singlemolecules. Fluorescence bead samples were drop cast onto coverglass fromsolutions of orange (100 nm), red (1 μm), nile red (2 μm), infrared (100nm) (Invitrogen, Thermo Fisher) and propidium iodide (10 μm) (BangsLaboratories, Inc.) beads. The IR bead solution formed crystals as seenin FIGS. 2B-C.

B4c) Data Analysis

Lifetimes were computed by both ratiometric calculation from imageintensities and by time-trace fitting. In ratiometric calculation, anumerically generated lookup table is used to convert between themeasured ratio and estimated lifetime according to the equations in thetext and the pre-characterized IRF. Due to our specific t_(d) andGaussian gate pulse in FIGS. 3A-C, lifetimes below 1.1 ns are redundantwith those above 1.1 ns in the numerical conversion. We report thelarger lifetime value. In timing trace calculation, fitting by leastsquares was used to estimate lifetime. The PC applies a time-varyingretardance to linearly polarized input as δ=2πr₆₃Vn_(o) ³/λ, where thebirefringent phase shift δ is determined by the applied voltage V,ordinary index of refraction n_(o), and the longitudinal electro-opticcoefficient r₆₃. Transmission in the parallel and perpendicularbeamsplitter channels is T_(s)=sin² (δ/2) and T_(p)=cos² (δ/2). Lifetimecalculations account for the imperfect gating efficiency of the Pockelscell as captured in the IRF. In FIGS. 2A-C, 3A-C, and 5A-D, a constantIRF is assumed across a conservative FOV. This may causeposition-dependent lifetime errors. In FIGS. 4A-C, spatial variation ismore apparent due to large FOV and is included in lifetime calculations.A beamsplitter in the microscope filter slider allows rapid switchingbetween fluorescence and frosted glass IRF calibration. IRF calibrationmay also be performed with a short lifetime dye.

Single-molecule gated and ungated intensities were determined by summingN_(p) pixels corresponding to each molecule region of interest afterbackground subtraction. Error bars in FIG. 3C account for shot noise inthe gated (G) and ungated (U) frames and for the background standarddeviations σ_(G) and σ_(U) in the ratio SE σ_(R) as

$\sigma_{R} = {\frac{G}{U}{\sqrt{\frac{1}{G} + \frac{1}{U} + {N_{p}\left( {\frac{\sigma_{G}^{2}}{G^{2}} + \frac{\sigma_{U}^{2}}{U^{2}}} \right)}}.}}$Background is the dominant error term here combining background signalwith a high camera dark current. Lifetime estimation accuracy for anideal two-bin PC gate is given by

$\frac{\sqrt{N}\sigma_{\tau}}{\tau} = {\frac{\tau}{\tau_{d}}{\sqrt{e^{\tau_{d}\text{/}\tau} - 1}.}}$The Cramér-Rao bound for n-bin lifetime estimation in a fixed timewindow of width T may be directly calculated from a multinomialprobability distribution. Fixed window bounds in FIGS. 6A-B were foundby setting T=n×t_(rt) for n round trips. In the large n limit, thesebounds demonstrate the performance of TC-SPC. The photon normalizedCramer-Rao bound for n bins is

$\frac{\sqrt{N}\sigma_{\tau}}{\tau} = {\frac{n\;\tau}{\tau}{{\sqrt{1 - e^{{- T}\text{/}\tau}}\left\lbrack {\frac{e^{\frac{T}{n\;\tau}}\left( {1 - e^{{- T}\text{/}\tau}} \right)}{\left( {e^{\frac{T}{n\;\tau}} - 1} \right)^{2}} - \frac{n^{2}}{e^{T\text{/}\tau} - 1}} \right\rbrack}^{{- 1}\text{/}2}.}}$C) Further VariationsC1) Modulator Configurations

As indicated above, various optical modulator configurations arepossible in addition to the example of FIG. 1A. Here FIG. 8A shows apolarization modulator 104 disposed between polarizers 120 and 124. FIG.8B shows polarization modulator 104 disposed between an input polarizer120 and an output PBS 106. FIG. 8C is similar to the example of FIG. 1A,except that two polarization modulators 104 a, 104 b are employed. Inthis case, the drive signals applied to the two modulators 104 a, 104 bcan be the same or they can be different.

These modulator configurations may also include double-pass variationswhere there is a mirror after the polarization modulator (in FIG. 8C forexample after modulator 104 a and/or 104 b) to reflect the light backthrough the same modulator(s). The beam-splitter before the modulator(102) may then split the reflected, modulated light from the incidentlight and direct it to a detector array. This has the advantage ofrequiring half the voltage for a given modulator phase shift. Adisadvantage is that double-pass does not easily allow both outputchannels simultaneously.

The example of FIG. 8D shows a series arrangement of optical modulatorsused to provide n-bin imaging. Here modulator 802, PBS 804, modulator806, PBS 808, modulator 810 and PBS are arranged in series as shown.Gating inputs to modulators 802, 806, 810 are step functions at T1, T2,T3, respectively, as shown. The result is that cameras 814, 816, 818,820 get time bins 822, 824, 826, 828, respectively. This exampleprovides 4 time bins, but any number of outputs can be provided in suchan arrangement, and any modulation waveform may be applied.

FIG. 8E shows another example of a series arrangement of opticalmodulators. Here first subsystem 850 include a PBS 852 and a modulator854 and second subsystem 860 includes a PBS 862 and a modulator 864. Theindividual modulators in such series arrangements can be any of theabove-described modulator configurations, in any combination. Eachpolarization modulator in the series for example may receive N inputimages, apply any modulation, and then generate 2N image outputsfollowing a beamsplitter. One application would be improving temporaldynamic range of a system by having fast and slow modulation waveformsapplied on sequential modulators.

C2) Hyperspectral Configurations

The example of FIG. 9A is similar to the example of FIG. 8C, except thatwavelength separating elements 902 and 904 (e.g. prism, grating) areadded, and cameras 906, 908 are now explicitly shown. The example ofFIG. 9B shows wavelength splitting elements 922, 924, 926 (e.g.,dichroic mirrors coupling the optical outputs to any number of cameras932, 934, 936. Such added dispersive elements can be provided at some orall outputs of the modulator. This can provide simultaneous measurementof space, nanosecond time, polarization, and color for a wide-fieldimage. Such time-resolved hyperspectral imaging is especially valuablefor single-molecule fluorescence microscopy.

Optical modulators may be combined with wavelength-resolved elements torealize multi-dimensional or ‘hyperspectral’ modes of imaging inwide-field (FIGS. 9A-B). In a fluorescence microscope having a pulsedexcitation, for example, each emitted photon has the following degreesof freedom: time emitted, polarization, spatial coordinates, andwavelength. By combining all-optical modulation based on polarizationwith wavelength-selective optics, all of these parameters may bemeasured simultaneously on a slow array detector.

In single-molecule spectroscopy and localization microscopy where thescene consists of sparse single-point emitters, a dispersive elementlike a prism, a diffraction grating, or a wedged filter stack may beinserted into output paths of the optical modulator. This allows forspectral information to be encoded as a linear streak or array ofemitter images. Similarly, wavelength splitting elements like dichroicmirrors may be used to split the output light into an array of colorchannels. This splitting method is compatible with wide-field images andnot restricted to sparse scenes. Absorptive color filters and sensorarray filters such as Bayer filters may also be employed.Multi-dimensional techniques allow for increased precision inmeasurements of Forster resonance energy transfer (FRET) betweenfluorophores by combining wavelength and lifetime channels. They alsoallow higher-dimensional imaging that can differentiate more individualfluorescent labels within a biological specimen.

C3) Resonant/Lock-in Operation

FIG. 10A shows an exemplary configuration for resonant/lock-in signalprocessing with this approach. Here the optical configuration is as onFIG. 8C. Modulator 104 a is driven by drive electronics 1010 at a firstsignal (ϕ1, ω1). Modulator 104 b is driven by drive electronics 1020 ata second signal (ϕ2, ω2). The resulting image at camera 1002 is at thefirst signal, and the image at camera 1004 is out of phase with camera1002. Similarly, the resulting image at camera 1006 is at the secondsignal, and the image at camera 1008 is out of phase with camera 1006.

FIG. 10B shows two examples of incident waveforms for lock-in signalprocessing: a sequence of fluorescence decays 1022 and a sinusoid 1024.More general periodic input waveform shapes are also possible.Modulation is applied to PC to demodulate fast varying waveform ontoslow detector arrays to estimate shape parameters. This configurationcan be regarded as a lock-in camera allowing de-modulation and fullvector measurement on a slow bandwidth detector by using two modulationphases.

Sinusoidal modulation enables estimation of waveform shape parameters inthe frequency domain. Our technique can implement either homodyne orheterodyne detection for wide-field images on standard camera sensors.An example is shown in FIGS. 10A-B. A polarizing beam splitter (e.g.,106, 108) after a polarization modulator (e.g., 104 a, 104 b) generatestwo intensity images. One of these corresponds to the convolution of theincident intensity waveform with the modulation. The second correspondsto the convolution of the incident intensity waveform with the inverseof the modulation. For a sinusoidal modulation, this means that the twooutputs have modulation phase shifted by 180 degrees. Just as fortime-gated waveform estimation, a significant advantage of our approachis the simultaneous acquisition of gated and ungated images. Here thismeans 0 and 180 degree phase images. This allows normalization ofincident light intensity and high speed, single-frame acquisition.

Frequency domain fluorescence lifetime estimation by homodyne is awell-known technique. Current wide-field approaches use either gatedoptical intensifiers or on-chip multi-tap modulated camera sensors toimage in the frequency domain. These have significant disadvantages inefficiency, cost, and speed. Our approach instead allows for all-opticaldemodulation of the fluorescent lifetime signal.

When a frequency modulated excitation is applied to a fluorescent scene,the fluorescence response can be characterized by its phase shiftrelative to the excitation and its modulation depth. Mathematically thisis usually described in terms of the sine and cosine Fourier transforms,G(ω) and S(ω) respectively, of the received light intensity. G and S arerelated to phase θ and modulation depth M of the response in thefollowing equations. They are often combined to allow phasor plotanalysis of fluorescence decays.

$\theta = {\tan^{- 1}\frac{S}{G}}$ $M = \sqrt{S^{2} + G^{2}}$Our techniques may produce a number of intensity outputs having adefined modulation phase and frequency. FIG. 10A shows four outputs (tocameras 1002, 1004, 1006, 1008), two from each single PC modulatorshifted 180 degrees. If a mono-exponential decay is assumed and theresponse light phase shift is pre-calibrated then only a single exposurefrom one modulator having two phases of 0 and 180 degrees on thedetector is required (I_(0,ω1) and I_(180,ω1)). This provides amodulation depth estimate for lifetime.

$M = {\sqrt{\frac{I_{{\phi 1},{\omega 1}} - I_{{{\phi 1} + 180},{\omega 1}}}{2\left( {I_{{\phi 1},{\omega 1}} + I_{{{\phi 1} + 180},{\omega 1}}} \right)}}\mspace{14mu}\left( {{single}\mspace{14mu}{frame}\mspace{14mu}{measurement}} \right)}$Phase of the response may similarly be measured by fitting multiplediscrete samples with each having a different modulator drive phase inanalogy to time-domain delay traces. A separate possibility is the useof multiple modulations each having a different drive phase. This allowsestimation of phase directly from four intensity outputs, for example

${\theta = {\tan^{- 1}\left( \frac{I_{0,{\omega 1}} - I_{180,{\omega 1}}}{I_{270,{\omega 1}} - I_{90,{\omega 1}}} \right)}},$and more generally full vector measurements of a periodic signal.

The phase and modulation depth provide two separate lifetime estimatorsbelow. Both may be compared, e.g. in phasor plots, to better estimatemulti-exponential lifetimes. Frequency domain estimation may approachthe same photon sensitivity limits as time-domain estimation.

$\tau_{phase} = {{\frac{1}{\omega}\mspace{14mu}\tan\mspace{14mu}\theta\mspace{14mu}\tau_{modulation}} = {\frac{1}{\omega}\sqrt{\frac{1}{M^{2}} - 1}}}$

Frequency domain operation realizes an imaging lock-in detector whereevery pixel of the imaging array detector is performing a separatelock-in or demodulation process analogous to a single lock-in amplifier.Two phase shifts may be combined to make a full measurement of a complexphasor acquiring both the in-phase and quadrature components. This maybe easily accomplished simultaneously by either having two modulatorsdriven with different phases or by optically introducing a phase shiftto some of the imaging beams using a retarder or waveplate.

Another possibility is the use of modulation frequency slightlydifferent from the illumination input to perform heterodyne detection.Slow beat frequencies may be detected on a slow camera chip for example.Similarly, series modulators could be driven with different drivefrequencies or incommensurate phases.

In addition to the unique requirements of Pockels cells being suited forwide-field imaging, high frequency operation presents its ownchallenges.

1) A high voltage AC voltage may need to be applied to the crystal. Anideal method for driving the PC is thus incorporating it into a resonantelectronic circuit like an LC tank circuit where it acts as a capacitor.This circuit should have a high Q-factor to enable practical driveelectronics.2) Heating of the Pockels cell crystal due to dielectric losses or ofthe Pockels cell electrodes due to resistive loss may require measuresfor active cooling. The crystal may be actively cooled by mounting itonto a cooled plate or by cooling its metal electrodes (e.g. intransverse field modulator geometries), by immersion in a dielectriccoolant (flowing or for static heat conduction), or by sandwiching alongitudinal modulator between heat conductive but optically transparentplates. Such plates could be made of glass, transparent ceramics, orsapphire for example and could connect to a heat-sink or thermal controlunit.C4) Charged Particle Detection

FIGS. 11A-C shows application of this work to charged particledetectors. These particle detectors (e.g., electron cameras) oftenconvert particle flux to a highly non-linear optical response by using aphosphorescent screen or a scintillating crystal. The resultant“fluorescence” decay waveform shape can be measured by our method inorder to provide temporal information about when a charged particle hitsthe detector by fitting a known lifetime. This allows fortime-correlated charged particle counting in a wide-field image with10's-1000's of simultaneous, spatially-resolved hits. This capabilitydoes not exist with current detectors and will allow new types ofmeasurements in electron microscopy.

FIG. 11A schematically shows three particles (#1, #2, #3) hitting aphosphor or scintillator 1104 at different positions and times. Here1102 is an optional multiplier or intensifier, 1106 is an opticalmodulator unit as described above, and 1108 is the camera. The differingtimes are schematically shown on FIG. 11B. Phosphor/scintillator 1104generates a burst of light having a long decay (80 ns for P47) and largephoton number (10{circumflex over ( )}6 for microchannel platemultipliers). The plot of FIG. 11C assumes some transit time spread(TTS) for a multiplier stage and an 80 nanosecond decay constant, andplots time estimation accuracy for an ideal step gate and a step gatehaving a long rise-time and 90% efficiency. Given large photon number Nfor microchannel plate+phosphor, this enables sub-100 picosecond timingresolution.

Time-correlated charged particle detectors have similar limitations totime-correlated single photon counters. Existing techniques combinemicrochannel plate electron multipliers with one or more anodes made ofcrossed-wire delay lines. A particle hit produces a burst of electronsfrom the microchannel plate which is spatially localized on thecrossed-wire anode based on pulse delay times in each line. Thisapproach is complex and limited to only a few simultaneous particle hitsand few megahertz count rates (low throughput). Our optical methodprovides an efficient alternative by using a scintillator or phosphorscreen to produce a phosphorescence or fluorescence decay waveform fromeach hit (FIGS. 11A-C). In this case, the lifetime of the decay waveformis known a-priori and the fitting of the phosphorescence waveform shapecan provide the time when the particle hits the screen.

For wide-field time-domain FLIM, an ideal gate can estimate lifetimewith shot-noise limited accuracy as described in the following equation:

${\sqrt{N}\sigma_{\tau}} = {\frac{\tau^{2}}{t_{d}}{\sqrt{e^{t_{d}\text{/}\tau} - 1}.}}$If the lifetime is instead known, then particle hit time may besimilarly estimated with shot-noise limited sensitivity as √{square rootover (N)}σ_(t)=τ√{square root over (e^(t) ^(d) ^(/t)−1)}. Microchannelplates at high gain may generate>10⁶ photons per pulse, allowing veryhigh temporal resolution in the estimation—even approaching the tens ofpicosecond jitter of the electron multiplier. Having access to bothgated and un-gated images is critical to enable normalization forvariable gain from each particle event. Typical transit time spreads ofMCP detectors are approximately 300 picoseconds with pulse jitters inthe 10's of picoseconds.

A time-correlated spatial detector for particles could be used inelectron microscopy to record high resolution space and time informationfor each imaging electron. For example, it might find use in ultrafasttransmission electron microscopes (UTEMS) or other electron microscopesand ultrafast diffraction experiments having pulsed or laser-triggeredemission sources. Further, such a detector could allow new imaging modesfor electron energy loss spectroscopy (EELS) where energy loss due toinelastic scattering in the sample results in a change in the electron'svelocity and arrival time. It can similarly enable the removal ofchromatic effects due to varying source energies in low-energy electronimaging systems for example low energy electron microscopes (LEEM) andphotoemission electron microscopes (PEEM)). This camera may further actas a quantum detector, enabling measurement of position and momentumcorrelations and detection of multi-particle coincidences.

The capability to measure >10 simultaneous hits is unique to ourtechnology, and extension to >10000 simultaneous hits is possible. Otherapplications include use in mass spectrometry for ion time-of-flightdetection, ion momentum spectroscopy experiments (e.g. cold targetrecoil ion momentum spectroscopy—COLTRIMS), and even single-photontime-correlated detection using image intensifier tubes.

C5) Endoscopic Applications

Wide-field optical modulators are promising for clinical fluorescencelifetime systems. Imaging of fluorescence and tissue auto-fluorescencecan provide an indicator for various disease and bio-markers. Use ofendoscopic, arthroscopic, or macro imaging systems in a clinical settingas the front-end for the optical modulator can allow for improvedidentification of diseased tissue and surgical margins. For example,FLIM allows measurement of NADH/NAD(P)H in cells as a marker ofmetabolism. This can provide an optical signature for cancerous tissue.Multi-spectral FLIM combining lifetime and wavelength dimensions canalso be a valuable diagnostic tool.

High speed acquisition and rapid lifetime calculation enabled by oursingle-frame method is especially valuable, as it allows real-timedisplay of fluorescence lifetime images and video-rate observationduring a medical procedure or operation.

Endoscopic systems may interface flexible optical fiber bundles,multi-mode optical fibers, and/or GRIN optics to the modulator unit(s).Relay lens systems may also be used such as in rigid arthroscopes.

The invention claimed is:
 1. Apparatus for providing time-resolvedoptical imaging, the apparatus comprising: a wide field opticalintensity modulator; one or more 2-D detector arrays; imaging opticsconfigured to image incident light onto the one or more 2-D detectorarrays through the wide field optical intensity modulator; wherein atemporal bandwidth of the optical modulator is greater than a temporalpixel bandwidth of the one or more 2-D detector arrays; a processorconfigured to automatically determine one or more waveform shapeparameters of the incident light by analyzing signals from the one ormore 2-D detector arrays vs. an input modulation applied to the opticalintensity modulator; wherein the one or more waveform shape parametersof the incident light are determined on a pixel-by-pixel basis of theone or more 2-D detector arrays.
 2. The apparatus of claim 1, whereinthe wide field optical intensity modulator comprises a wide fieldoptical polarization modulator disposed between a first polarizer and asecond polarizer so as to convert polarization modulation to intensitymodulation.
 3. The apparatus of claim 1, wherein the wide field opticalintensity modulator comprises an input polarizer followed by a widefield optical polarization modulator followed by a polarizing beamsplitter, wherein the polarizing beam splitter provides a first outputto a first of the 2-D detector arrays and provides a second output to asecond of the 2-D detector arrays, whereby polarization modulation isconverted to intensity modulation of the first and second outputs. 4.The apparatus of claim 3, wherein the one or more waveform shapeparameters includes an exponential decay time, wherein the inputmodulation is a step function, and wherein the exponential decay time isdetermined by analysis of single-frame signals from corresponding pixelsof the first 2-D detector array and the second 2-D detector array. 5.The apparatus of claim 1, wherein the wide field optical intensitymodulator comprises: an input polarizing beam splitter having a firstoutput and a second output; a wide field optical polarization modulator(PM) configured to receive the first and second outputs in parallel andto provide corresponding first and second PM outputs; a first outputpolarizing beam splitter configured to receive the first PM output andto provide a third output and a fourth output; a second outputpolarizing beam splitter configured to receive the second PM output andto provide a fifth output and a sixth output; wherein the third outputis provided to a first of the 2-D detector arrays; wherein the fourthoutput is provided to a second of the 2-D detector arrays; wherein thefifth output is provided to a third of the 2-D detector arrays; whereinthe sixth output is provided to a fourth of the 2-D detector arrays. 6.The apparatus of claim 1, wherein the input modulation is a pulse havingan automatically adjustable time delay t_(d) after an optical excitationprovided to a scene, and wherein the one or more waveform shapeparameters include data points of detector array signals vs. time delay.7. The apparatus of claim 1, wherein the input modulation is selectedfrom the group consisting of: a step function, a sampling pulse, andperiodic modulation for lock-in detection.
 8. The apparatus of claim 1,wherein the wide field optical intensity modulator includes alongitudinal Pockels cell having a direction of optical propagation andan applied electric field direction that coincide.
 9. The apparatus ofclaim 1, wherein the imaging optics include a multipass optical cavityhaving a cavity round trip time, wherein the multipass optical cavity isconfigured to provide optical time resolution according to multiples ofthe cavity round trip time.
 10. The apparatus of claim 1, wherein theincident light is a periodic signal that is responsive to a periodicexcitation of a scene being viewed, and wherein the wide field opticalintensity modulator is resonantly driven synchronously with respect tothe periodic signal.
 11. The apparatus of claim 1, wherein the opticalintensity modulator includes two or more optical modulators havingidentical or different input modulation signals.
 12. The apparatus ofclaim 1, wherein the imaging optics is configured to view a scene. 13.The apparatus of claim 12, wherein an optical response of the scene toan excitation provides the incident light.
 14. The apparatus of claim13, wherein the optical response of the scene is a nonlinear response.15. The apparatus of claim 13, wherein the wide-field optical intensitymodulator is driven with a modulation signal having a controllable delayafter the excitation.